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Abstract 

A recent dynamic mean-field theory for sequence processing in fully 
connected neural networks of Hopfield-type (Diiring, Coolen and Sher- 
rington, 1998) is extended and analyzed here for a symmetrically di- 
luted network with finite connectivity near saturation. Equations for 
the dynamics and the stationary states are obtained for the macro- 
scopic observables and the precise equivalence is established with the 
single-pattern retrieval problem in a layered feed-forward network with 
finite connectivity. 
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1 Introduction 



A path- integral approach [1, 2] has been successfully applied already some 
time ago [3] to study the dynamics near saturation of the Hopfield model 
[4] with a symmetric Hebbian learning rule that favors the retrieval of single 
patterns. The results for the stationary states were found to be those ob- 
tained by means of the replica method in equilibrium statistical mechanics 
[5] . The performance of networks trained with a sequence of stored patterns 
has been of interest over some time [6 — 13]. for both biological and artifi- 
cial neural networks [14], and the path- integral approach has been applied 
to study the single-pattern stationary states in a mean-field dynamics near 
saturation for a fully connected recurrent network trained with a sequence 
of stored patterns [15]. 

More recently, the transients in that network have been investigated in a 
statistical neurodynamics approach, which was shown to be equivalent to the 
path-integral method [16]. These methods are particularly suitable to handle 
dynamical systems with asymmetric interactions, as in the case of sequential 
patterns, due to the absence of detailed balance where equilibrium statistical 
mechanics cannot be used. 

Explicit equations were obtained for the retrieval overlap with single pat- 
terns in a sequence, for the response function to an external field and for the 
two-state correlation function [15, 16]. By means of exact analytic work, sup- 
ported by numerical simulations, it has been found that the critical storage 
capacity in the fully connected recurrent network for single-pattern retrieval 
in a sequence is cic = 0.269 [15]. This is the same as the critical storage 
capacity for a different problem in another network architecture, namely, the 
usual single-pattern retrieval problem in the layered feed-forward network 
[17]. It has been pointed out that this is due to the equivalence between the 
two models in the absence of synaptic noise (T = 0), and that this equiv- 
alence does not extend beyond the T = limit. The performance of the 
network turns out to be limited by a non-local (in time) Gaussian noise in 
the effective single-spin problem [15]. 

The equilibrium behavior of Hopfield-type networks with finite random 
decay (dilution) of synapses is of interest for biological [18] and artificial neu- 
ral networks and has already been considered for the single-pattern retrieval 
problem in systems with or without microscopic detailed balance [17, 19 — 21]. 
Dense networks with connectivity c = 0(1) share most of the features of fully 
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connected networks and have a non-trivial dynamics, in contrast to the ex- 
tremely diluted network [22]. 

We discuss here the extension of the sequence processing dynamics of 
Diiring and al. to finite synaptic dihition and study the precise relationship 
with the single-pattern dynamics and the asymptotic stationary state of the 
layered feed-forward network. This is a network that consists of identical 
layers of N non-interacting units each, with synaptic interaction J^j between 
units j and i in layers I and I + 1, respectively. 

The outline of the paper is the following. In Sec. 2 we present the model 
and derive the connected generating functional. In Sec. 3 we discuss the 
effective single-site problem in dynamic mean-field theory and present the 
specific equations for the macroscopic observables, including the stationary 
state. We discuss there the explicit relationship with the results for the 
layered feed-forward network and summarize our conclusions in Sec. 4. 

2 The model 

2.1 Defining relations 

We consider a network of N Ising neurons in a microscopic state cr(t) — 
{ai{t), . . . ,(Tj^{t)}, at the time step t in which each (Tj(t) = ±1, that is 
updated simultaneously according to the alignment of each unit to its local 
field 

h^{t)=J:Jia,ii)+e,it) , (1) 
j 

following a stochastic dynamics with transition probability 

Wi[ai{t + l)\ai{t)] = ^[1 + ai{t + 1) tanh{pai{t)hi{t))] , (2) 

ruled by the noise-control parameter /5 = T~^. The dynamics is a determin- 
istic one when T — and it is fully random when T — oo. In the former 
case, the state of a spin at a given time is completely determined by the sign 
of its local field following the relation (Tj(t + 1) = sgn[/ij(t)]. Here, Jfj is the 
synaptic coupling to other neurons and 6i(t) is an external field introduced 
in order to study the response of the network to an external stimulus. Thus, 
given the probability p[cr(0)] of an initial state, Eq.(2) determines the path 
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probability p[cr(0), . . . , cr{t)] that takes the system to the state a{t) through 
a Markov chain. 

A macroscopic sequential set of p = aN independent and identically 
distributed random cyclic patterns ^'^ = (^f , . . . , ^^(;), /x = 1, . . . ,p, such that 
^p+i _ ^1 g^]^^ each = ±1 with probability |, is assumed to be stored in 
the network according to the learning rule 

4 = ||Er^e;, (3) 

and we are interested in the behavior of the network near saturation, that 
is, for finite a. The synaptic interactions are assumed to be symmetrically 
diluted by means of the set of couplings {cjj}, with Cij = cji taken from a 
set of independent identically distributed random variables, which are the 
same for every time step, such that Cij = 1 with probability c and zero 
with probability 1 — c, while cu = 0. Thus, the average [cij] = c becomes 
the connectivity of the network and when c — 1 one has the usual synaptic 
coupling for the fully connected case. We consider a dense network, with 
c = 0(1), and take cN >> 1 allowing eventually for a vanishingly small 
c after taking the thermodynamic limit in the final results of the dynamics. 
Once the network has a finite connectivity it is expected to have a non-trivial 
dynamics which is different from that of a purely random network as long 
as (3 ^ 0. Thus, one has to ensure first a finite connectivity and we do this 
next. 

2.2 The connected generating functional 

Since the matrix J'^ = {•^fj} non-symmetric, one has to resort to a dy- 
namical procedure to study the time evolution and the stationary states of 
the network and we follow previous works using the generating functional or 
path- integral method [15, 16]. We introduce a connected generating func- 
tional 

ZM= E (p[c^(0),...,tT(t)])e,,e-S.<*^(^)-^(^) , (4) 

(T(o),...,cr(t) 

where (• • .)c, denotes the path probability for the connected network ob- 
tained by averaging p[cr(0), . . . , cr(t)] over the set {cij} and, as usual, x.y = 
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^iXiUi. Here, il^iis) = .... il-){t — 1)} is introduced and afterwards set 

to zero in order to generate state averages over the path probabihty. Using 
an integral representation of unity to account for the defining relation for the 
set of local fields given above, by means of an auxiliary set {hi{s)} 

[15], we can separate the configurational average over the set {c^} and obtain 

Zc{^) = E P[<^)] [ {dhdh} 
<r(o),...,<T(t) 

nexp{/3(T(s + l).h(s) - 5^1n2cosh(/3/ii(s)) 

s<t i 

+ zh{s).[h{s) - e{s)] - ztl^{s).cr{s)}{e-^^^^^-'''^^^%, , (5) 

where {dhdh} = Hi n.<t[d/ii(s)d/ii(s)/27r]. 

We proceed next to calculate the configurational average along similar 
lines to those used in statistical mechanics of diluted spin-glass and neural 
network models [23, 21] and write 

(e-M^-'-'^'^i^A = l[{l + c{eM-^hi{s)J^j^j{s)]-^)} 

= exp[^ln(l + cr/ij(s))] , (6) 

where the coupling J^j = Jf- for = 1 and riij{s) = exp[— JjjCrj(s)] — 1. 
Keeping in mind that we want results for finite c and noting that Jij = 
0{^Ja/cN) is small, we use first rjij as an expansion parameter up to second 
order and then expand in J^j, to that order, and obtain 

^^-.h(.).j^.(s)^^^^ = exp{-ih(.).J^r(.) - ic(l - c) UhimMs)]'} . (7) 

The couplings on the right J = {Jij}, with J^j = J^fi^i'^^^j 7 now 
the synapses for the fully connected network with sequential patterns and 
the second term in the argument of the exponential contains the full effect 
of the dilution. 

Before doing the average of the exponential over the patterns following 

the standard procedure below, one may replace here the pattern dependence 
of the last term, up to terms of higher order which are irrelevant in mean- 
field theory, by its average over the pattern distribution becoming thereby 
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pattern independent. In this way, the state cr(s) becomes decoupled from 
the auxihary field h(s) and, noting that Y^i^^ii^) = obtain 

/^-ih{s).J(T(s)\ ^ g-ih(s).JO-(s)g-A2h2(s)/2 ^ /gx 

\ / dj 

where = a{l — c)/c and h^(s) = J2i h^is). Note that is of order 1/VN, 
the local field /ij and hi are of order one and, consequently, the argument of 
both exponents is, as one would expect, of order N. Incidentally, the result in 
Eq.(8) is precisely the average (exp(— ih(s).J^-'^-^cr(s)))|5.^.| over the pattern- 
independent set {Sij} of independent Gaussian random variables with mean 
zero and variance A^, in which the effective coupling J^J-^ = Jij + 5ij. Thus, 
as in the case of the equilibrium behavior of networks, the finite symmetric 
dilution in the dynamics amounts to the addition of a static Gaussian noise 
to the synaptic couplings of the fully connected network. [19]. 

This completes the derivation of the formal expression for Zc{^), in which 
the connected path probability takes the form 

(p[o-(0),...,o-(t)]),,^. = p[c7(0)] /{dhdh}nexp{M5 + l)-h(s) 

s<t 

- ^ In 2 cosh{l3hi{s)) + zh(s).[h(s) - 6>(s)] 

i 

- ih{s).J^{s) - lA'h'is)} . (9) 

Clearly, the auxiliary variable h has the role of relating microscopic physical 
quantities with each other and this will show up next in constructing the 
macroscopic observables. 



2.3 The macroscopic observables 

The macroscopic dynamic observables depend on the problem one wants to 
study. If one is interested in the retrieval of a single pattern, say ^- at each 
time step s, the case to which we restrict ourselves in this work, one assumes 
that that pattern is 'condensed' which means that the overlap with the state 
of the network, <t(s), at that time step is of 0(1) and that all other overlaps 
are of 0{N-2). The non-condensed overlaps introduce a stochastic noise 
into the dynamics which may be thought as a quenched disorder. One then 
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expects that the relevant macroscopic variables for single-pattern retrieval 
are the overlap m7v(s), the single-site response and correlation functions, 
Gjv(s,s/) and Civ(s,s/), respectively, defined and given, for large but finite 
AT, by 

.W^EM-i-^ES^. (10) 

G«(«. = ^ E = ' l.m - E . (11) 

C.(s, s,) . i j:iMs,)) = - lim 1 E , (12) 

where the brackets now denote the thermal averages over the connected path 
probability given by Eq.(9). Note that, due to the form of Zc(i/'), Eq-(5), the 
response function is given by the i/'(s) limit of —iN^^ J2i{o'i{s)hi{s/)). In 
principle, there could be other observables, as kN{s) = N^^ J2i hi{s) and 
Qn{s, sf) = N"^ J2i hi{s)hi{sf), but they do not represent physical quantities 
so that one would expect that they do not show up in mean-field theory. 
Indeed, it has been found that the saddle-point solutions for these observables 
turn out to vanish in mean- field theory [15]. 



3 Dynamic mean-field theory 

As usual in neural network studies, one can only deal with a finite number of 
condensed patterns. Having assumed that at each time-step a single pattern 
is condensed, a finite sequence of patterns with fi < t will be condensed in 
fixed time t. Thus, the dynamical mean-field theory will be restricted to 
finite time scales although we keep the limit — > cxo. In that limit, the 
theory should be self-averaging and we are then left with an explicit average 
of the generating functional over the non-condensed patterns, Zc(i/')]. Noting 
that only the part of the exponential depending on J on the right in Eq.(9) 
is pattern dependent, one can follow the steps in ref. 15 to do this average 
and the relevant macroscopic observables in the dynamic mean-field theory, 
in the limit N ^ oo, become 

m^s) = lim 1 E = ^ lim Hm 1 ^ C!^^ , (13) 
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G(s, sf) = -i lim - y {ai(s)hi(sf)) = i lim lim - V ^ ^ ,^"[^] , ,(14) 



C(s,sf) = lim ly ((Ti(s)(7i(s/)) = - lim lim 1 V -A44^1- . (15) 
^' ^ iv^ooiVY V'-o^-ooiV Y ^^i(s)^^i(s') 

3.1 Single-site relations 

An explicit expression is then obtained in the form of an effective single- 
site normalized average for any function /[{cr}] of the states of the diluted 
network with finite connectivity, denoted as (/[{c}])*, for p — aN in the 
limit ^ oo. The normalization is such that the average is unity when 
/[{a}] = 1, and the result becomes independent of the specific site after 
the gauge transformation a{s) — >• cr(s)^| and h{s) —>■ h{s)^^~^^ [15]. This 
expression reads, 

if [WW = E [{dhdh}p{a{0))f[{a}]eMT.[f3<^is + ms) 

a{0),...,cr{t) s<t 

- ln2cosh{(3h{s))] + iY^h{s)[h{s) - e{s) - m'{s)] 

s<t 

-lay h{s)D{s,sf)hisf)} , (16) 

^ s,sl<t 

in which {dhdh} = Y[s<t[dh{s)dh{s) /27i] and all quantities are single-site 
quantities. Here, 

1 - c 

D{s, sf) = R{s, sf) + Ss,s, , (17) 

is the covariance D{s, sf) = {v{s)v{sf)) of the zero-average Gaussian random 
noise v{s) in the effective local field 

h(s)^m%s) + e{s) + ^/av{s) , (18) 

as can be seen readily by integration over h. Thus, the effect of the dilution 
appears as a local term (in time) contributing to the variance {v'^{s)) of the 
noise. It will be seen below that, precisely because of its local character, the 
dilution term has a crucial role in determining the stationary state of the 
network. The non-local contribution is that in the covariance of the noise. 
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R{s, s/), for the fully connected network and this is given by the recurrence 
relation [16] 

R{s, sf) = C{s, sf) + G{s, s - l)G{sf, sf - l)R{s - 1, s/ - 1) . (19) 

Combining Eqs.(17) and (19) with the appropriate average over the states 
given by Eq.(16), one obtains the evolution equations for the macroscopic 
observables. 

3.2 Equations for the dynamics 

The single-site equations for the time evolution of the macroscopic observ- 
ables are then given by 



G{s,sf) = PS,,,^^{1- lidvdwye''" '^-'^'^-^'^ 

X tanh^ (3[m{s - 1) + e{s - 1) + ^/av{s - 1)]} , (21) 



where D is the covariance matrix of elements D{s, sf). Note that the argu- 
ments of the tanh are /3 times the local field at time s or sf. Prom here on 

we drop the pattern index in the overlap m^(s) with the understanding that 
m(s) means the overlap with the pattern in the sequence that corresponds 
to that time step. 

It has been pointed out before, for the fully connected network [15], that 
the response function differs from zero only for changes of state one time step 
after the perturbation in the external field and that, consequently, stationary 
macroscopic states should be reached in finite timescales. Clearly, that is still 
the case for the diluted network. 




X tanh/?[m(s - 1) + 9{s - 1) + ^/av{s - 1)] , 



(20) 




X tanh /3[m(s - 1) + 0{s - 1) + \^v{s - 1)] 
X tanh/?[m(s/ - 1) + 9{sf - I) + ^/av{sf - 1)] 



(22) 
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The equations can be reduced by cxpUcit integration over w, noting that 
only the matrix element D{s — 1, <s — 1) is needed for the first two of them 
and we obtain 

m{s) ^ j Dz tanh (5[m{s - 1) + e{s - 1) + ^^^^(s - 1, s - 1)] , (23) 

G{s,s-l) = f3{l-jDz 

X tanh^ (3[m{s - 1) + 9{s - 1) + z^aD{s - 1, s - 1)]}{2,4:) 

where z is a Gaussian random variable with mean zero and unit variance 
and, as usual, Dz — The third equation becomes 

C{s,sf) = (27r|r>n|)-^/'|dze-^^-^"^ 

X tanh p[m{s - 1) + 9{s - 1) + ^/az{s - 1)] 

X tanh (3[m{sf - 1) + 9{s/ - 1) + ^/az{s/ - 1)] , (25) 

when s ^ St and C(s, s) — 1. Here, Du is the 2x2 submatrix of D involving 
the elements at times s — 1 and s/ — 1, while z is the vector with components 
z{s — 1) and z{si — 1). The diagonal elements of Du are given by 

D{s,s) = R{s,s) + {l-c)/c 

R{s,s) = l + G'^{s,s-l)R{s -l,s-l) (26) 

and only the off-diagonal elements depend on the correlation function, which 
enters the dynamics through Eq.(25). 

In the case where c—1, our equations, coincide with those for the tran- 
sients in the fully connected network [16]. In the extremely dilute limit c — > 0, 
the storage ratio a has to be replaced by the ratio p/cN, that is, the number 
of patterns per connected units, which means a/c. The equations for the 
overlap and the response function become then those for the extremely di- 
luted network, and the correlation function vanishes in that limit. Given an 
initial value for R{s, s), Eqs.(23),(24) and (26) form a closed set of equations 
for the time evolution of the macroscopic observables of primary interest. 
It is important to keep in mind that these quantities do not depend on the 
correlation function. 
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It may be noted at this point that, for general c, Eqs.(23),(24) and (26) 
are formally the same as the recurrence relations for the overlap and the 
stochastic noise in the layered feed-forward network with finite connectivity 
for the single-pattern retrieval problem [17]. The task in this problem is to 
recognize a given single pattern as the states evolve from one layer to the next 
one. The network model consists of N binary units placed on layers in which 
all units of a layer are updated simultaneously according to the alignment 
with their local field produced by the units in the previous layer and there is 
no feedback from any layer to previous ones. A set of independent random 
patterns, ■ ■ ■ j'CjvlO}) = li • • • iPj each = ±1 with probability |, 

is generated on every layer / independently of other layers, and the patterns 
are assumed to be stored in the network according to the learning rule 

4(i)-^t^ni+m(i) ■ (27) 

IJ.=1 

The local field at unit i on layer I may then be written as 

hS) = ^1(1 + l)m\l) + 9,(1) + z,{l) , (28) 

where the first term is the signal from the finite overlap with pattern say, 
m}{l) and 

^^(o = Eer(^+i)^'^(o (29) 

is the noise due to the remaining patterns with overlap m^{l). The recurrence 
relations for the overlap w}{l) and for the variance of the noise, D{l,l), are 
then given precisely by Eqs.(23), (24) and (26) if the time step is replaced 
by the layer index and writing G{1, 1 — 1) = P[l — q{l — 1))], where [17] 

q{l - 1) = j Dz tanh^ p[m{l - 1) + e{l - 1) + aD{l - 1,1 - 1)] (30) 

with 

R{1, = 1+ /9^[1 - q{l - l)fR{l - 1, ^ - 1) (31) 

and the relationship with D{l,l) given above. The variable q{l) is, clearly, 
the single-site average {ai{l)y in the layered network, where the brackets 
denote here the average with the distribution given by Eq.(2). 

Consider now the correlation function in the layered network. To ob- 
tain C(/,//) one needs to establish a recurrence relation for the covariance 



10 



{zi{l)zi{ll)) of the noise. Following a standard procedure [16, 17], one finds 
that, for a 7^ 0, the covariance is zero due to the independence of the patterns 
in different layers and, consequently, C{l,lf) — 5i^i,. 

3.3 The stationary state 

The stationary state of the network is obtained from the time-translation 
invariant limit cycle solutions 

m{s) = m , C{s, s/) = C{s - sf) , G{s, s/) = G{s - sf) (32) 

for the macroscopic observables. Then, R{s,s/) = R{s — s/) and D{s,s/) = 
D{s — sf) with R{s,sf) given by Eq.(19). The equations for the first two 
macroscopic observables in the stationary state become then 

j Dz tanh p[m + 9 + z^a{p+{l - c)/c)] , (33) 

q = J Dz tanh^ (3[m + 9 + z^J a{p + {I - c)/ c)] , (34) 
where q is given by G(l) = /3(1 — q). Here, 

is the stationary value of the variance in the internal field for the fully con- 
nected network. These are, again, the equations for the stationary state 
for the single-pattern retrieval problem in the layered feed-forward network 
with finite connectivity and with an effective Gaussian noise of variance 
p — a{p + {1 — c)/c). The results at T = have already been discussed, 
and it has been found that both, the critical storage capacity etc and the 
limiting overlap of the recalled pattern, are monotonic decreasing functions 
of the dilution [17]. 

To obtain the correlation function in the stationary state it is convenient 
to separate in Eq.(22) the persistent and non-persistent parts in R{t) [15], 

R{t) = r + R{t) , J-im ^(r) = , (36) 

which yields D{t) —r-\- D{t), where 

D(t) =%) + [(l-c)/c]5,,o (37) 
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also vanishes in that hmit. The separation of the persistent part in D{t) 
and a hnearization introduces a new Gaussian integration over a variable z 
of mean zero and variance ar, where r — qp and q — lim^^oo C(t) is the 
persistent part of C(r). This may be interpreted as the spin-glass order 
parameter. The remaining Gaussian random variable has covariance aD{T) 
and, in the limit r oo, v{t) and v{0) become uncorrelated. We are left 
with the variance aD{0), where 

D{0)^{l-q)p+^ , (38) 

and we obtain for the persistent part of the correlation function 

q = j Dz{j Dx tanh/3[m + 6 + z^faqp + a;y^a((l - q)p + (1 - c)/c)]}l 

(39) 

We consider next the solutions of this equation combined with the re- 
sponse function in the absence of recall for ^ = 0. This requires q^ which is 
always non-zero and vanishes at high T as 

g = a/5'(l + ^) + 0(/3^) . (40) 
c 

In the limit T — > 0, /3(1 — q) remains finite and this leads to p = 1 -|- Ijiia. 
when m = 0. Consider next the equation for q and note that it always has 
the paramagnetic solution g = in the absence of recall. In the limit T — > 0, 
we find 



q=\Dz erf ^ \ ^^^^^ ] , (41) 

^ 72[(l-g)p+(l-c)/c] 

for non-zero a, and analysis of this equation shows that, for any connectivity 
c < 1, the only solution is g = 0. In contrast, there is also the solution g = 1 
in the case of the fully connected network. We also conclude that there is 
no solution other than g = for finite T, regardless of the connectivity, in 
extension of earlier work [15]. Thus, the persistent part of the correlation 
function vanishes for all T in the sequence processing network when c < 1. 
In contrast, it is always zero in the layered network, as we saw above. 
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4 Summary and conclusions 



We extended the path-integral method to study the retrieval dynamics of 
consecutive single patterns in a sequence for a symmetrically diluted network 
near saturation in a dynamic mean-field theory. This is an exact procedure 
for an asymptotically large network and the close agreement with numerical 
simulations obtained in recent works [15, 16] seem to indicate that the theory 
accounts for the essentials of single-sequence processing. The main feature 
is the presence of an, in general, non-local Gaussian stochastic noise in time 
in the effective single-site problem. Only the local part of that noise enters 
in the equations for the dynamics and the stationary states for the overlap 
with a pattern and for the response function to an external field. 

The symmetric dilution of the synapses, which is equivalent to the addi- 
tion of a static noise to the coupling of the fully connected network, introduces 
a further local Gaussian noise which is relevant for the retrieval performance 
of the network. We have shown that the resulting equations for the dynamics 
and for the stationary states of the sequence processing network are precisely 
the same as those for the single-pattern retrieval problem in the layered feed- 
forward network, for all T. We also showed that the particular form of the 
resulting Gaussian stochastic noise is exactly the same in both systems for 
any finite connectivity. 

Nevertheless, the two systems are not fully equivalent. Indeed, although 
the two-time correlation function is zero in the layered network, there is a 
finite, persistent correlation q — 1 for the fully connected sequence processing 
network at T = [15]. However, we also showed that this finite value dis- 
appears as soon as there is a dilution of the couplings, restoring in that case 
the formal equivalence between the performance of both systems. We insist 
here that the task of the network and its architecture are different for both. 
In the case of finite T there is no persistent part of the two-time correlation 
function in both systems, regardless of the connectivity. 

Finally, all the results discussed here with the path-integral method also 
follow from the statistical neurodynamics approach. It would be interesting 
to study the effects of synaptic dilution on other tasks in a sequence pro- 
cessing network, like mixture-state retrieval, in which the overlap with more 
that one pattern in a sequence remains finite. It may also be interesting to 
introduce a dynamical dilution process that could account for deteriorating 
synapses. The random symmetric dilution used here is a static one that is 
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not altered in the evolution of the network. These are issues that will be 
studied in future work. 
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